Add FE tests w/ Robot Framework #38
Conversation
After submitting blackaryGH-37 as a potential fix for blackaryGH-35, I wondered if there was a way to add FE tests that would detect that sort of issue. I found that [Robot Framework] is a popular tool for this, and it has a [SeleniumLibrary] that can be used to interact with web pages. This commit adds a simple test case that opens a browser, inputs text into the `st_keyup` input field, and waits for the page to contain some expected text. The test case is run by a shell script that uses the `robot` command to run the test case. The test case is run by a shell script that uses the `robot` command to run the test case. This test fails with the code on `main` when using Streamlit 1.32.2, because of blackaryGH-35: ``` (.venv) abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (fe-tests-robot-framework) $ pip freeze | grep '^streamlit' streamlit==1.32.2 (.venv) abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (fe-tests-robot-framework) $ ./run-robot-tests ============================================================================== Robot ============================================================================== Robot.Test Cases ============================================================================== Interact with st_keyup | FAIL | Element 'css:iframe[title="st_keyup.st_keyup"]' not visible after 5 seconds. ------------------------------------------------------------------------------ Robot.Test Cases | FAIL | 1 test, 0 passed, 1 failed ============================================================================== Robot | FAIL | 1 test, 0 passed, 1 failed ============================================================================== Output: /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/output.xml Log: /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/log.html Report: /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/report.html ``` But it passes with the code in blackaryGH-37: ``` (.venv) abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (blackarygh-35-not-rendering-with-streamlit-1.32.2●) $ git branch fe-tests-robot-framework * blackarygh-35-not-rendering-with-streamlit-1.32.2 main (.venv) abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (blackarygh-35-not-rendering-with-streamlit-1.32.2●) $ git checkout -b blackarygh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework Switched to a new branch 'blackarygh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework' (.venv) abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (blackarygh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework●) $ git merge fe-tests-robot-framework Merge made by the 'ort' strategy. .gitignore | 5 ++++- run-robot-tests | 3 +++ tests/robot/resources/app_keywords.robot | 16 ++++++++++++++++ tests/robot/test_cases.robot | 16 ++++++++++++++++ 4 files changed, 39 insertions(+), 1 deletion(-) create mode 100755 run-robot-tests create mode 100644 tests/robot/resources/app_keywords.robot create mode 100644 tests/robot/test_cases.robot (.venv) abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (blackarygh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework) $ ./run-robot-tests ============================================================================== Robot ============================================================================== Robot.Test Cases ============================================================================== Interact with st_keyup | PASS | ------------------------------------------------------------------------------ Robot.Test Cases | PASS | 1 test, 1 passed, 0 failed ============================================================================== Robot | PASS | 1 test, 1 passed, 0 failed ============================================================================== Output: /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/output.xml Log: /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/log.html Report: /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/report.html ``` [Robot Framework]: https://robotframework.org/ [SeleniumLibrary]: https://robotframework.org/SeleniumLibrary/SeleniumLibrary.html
|
I love this! I've wanted to play with Robot Framework, but hadn't gotten a chance yet. A couple things I would like to see before this gets merged, which I should be able to do eventually but you certainly can do if you would like to:
|
with robotframework deps
See e1b6d3f. |
Yeah I agree that having CI to automate this will be great. Personally, I don't have a lot of experience with GitHub Actions. I recently set up a work project to run Robot Framework in CircleCi; it's probably fairly similar. The bigger issue I see that I think deserves some thought is that running it on every commit to your repo is not quite enough. In fact, in the case of #35, AFAICT that was not caused by any changes in your repo; rather it seems to have been caused by a new version of Streamlit being released that violated implicit assumptions in streamlit-keyup. So while running Robot Framework tests on every commit is a good thing, I think ideally it would also be run automatically when new dependencies are released, or perhaps just on a schedule like nightly or weekly, which is probably easier. Hmmm. I do wonder if this could be solved by having a set up with one of these bots like dependbot or such that notices new versions of dependencies and then creates PRs to upgrade to them. It also seems like a good thing to ensure that streamlit-keyup works with a wide range of Streamlit versions; not just the latest. So maybe that implies doing some kind of matrix build thing where you run the tests against multiple Streamlit versions and multiple Python versions. Not sure what the best way is yet, but I might be able to find a little time to send some PRs to experiment. |
After submitting GH-37 as a potential fix for GH-35, I wondered if there was a way to add FE tests that would detect that sort of issue. I found that Robot Framework is a popular tool for this, and it has a SeleniumLibrary that can be used to interact with web pages. This commit adds a simple test case that opens a browser, inputs text into the
st_keyupinput field, and waits for the page to contain some expected text. The test case is run by a shell script that uses therobotcommand to run the test case. The test case is run by a shell script that uses therobotcommand to run the test case.This test fails with the code on
mainwhen using Streamlit 1.32.2, because of GH-35:But it passes with the code in GH-37:
Cc: @blackary, @nathancooperjones